Natural Temporal Difference Learning

In this paper we investigate the application of natural gradient descent to Bellman error based reinforcement learning algorithms. This combination is interesting because natural gradient descent is invariant to the parameterization of the value function. This invariance property means that natural gradient descent adapts its update directions to correct for poorly conditioned representations. We present and analyze quadratic and linear time natural temporal difference learning algorithms, and prove that they are covariant. We conclude with experiments which suggest that the natural algorithms can match or outperform their non-natural counterparts using linear function approximation, and drastically improve upon their non-natural counterparts when using non-linear function approximation. Introduction Much recent research has focused on problems with continuous actions. For these problems, a significant leap in performance occurred when Kakade (2002) suggested the application of natural gradients (Amari 1998) to policy gradient algorithms. This suggestion has resulted in many successful natural gradient based policy search algorithms (Morimura, Uchibe, and Doya 2005; Peters and Schaal 2008; Bhatnagar et al. 2009; Degris, Pilarski, and Sutton 2012). Despite the successful applications of natural gradients to reinforcement learning in the context of policy search, it has not been applied to Bellman-error based algorithms like residual gradient and Sarsa(λ), which are the de facto algorithms for problems with discrete action sets. A common complaint is that these Bellman-error based algorithms learn slowly when using function approximation. Natural gradients are a quasi-Newton approach that is known to speed up gradient descent, and thus the synthesis of natural gradients with TD has the potential to improve upon this drawback of reinforcement learning. Additionally, we show in the appendix that the natural TD methods are covariant, which makes them more robust to the choice of representation than ordinary TD methods. In this paper we provide a simple quadratic-time natural temporal difference learning algorithm, show how the idea of compatible function approximation can be leveraged Copyright c © 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. to achieve linear time complexity, and prove that our algorithms are covariant. We conclude with empirical comparisons on three canonical domains (mountain car, cartpole balancing, and acrobot) and one novel challenging domain (playing Tic-tac-toe using handwritten letters as input). When not otherwise specified, we assume the notation of Sutton and Barto (1998). Residual Gradient The residual gradient (RG) algorithm is the direct application of stochastic gradient descent to the problem of minimizing the mean squared Bellman error (MSBE) (Baird 1995). It is given by the following update equations: δt = rt + γQθt(st+1, at+1)−Qθt(st, at), (1) θt+1 = θt − αtδt ∂δt